This page provides complete documentation of our E155 final project: an Invisible Drum Set that enables playing virtual drums through air gestures detected by an IMU sensor.
Abstract
This project aims to adapt the existing Invisible Drum Set—originally implemented with an Arduino Nano and BNO055 IMU sensor—onto the E155 hardware platform, consisting of an STM32 MCU and UPduino FPGA. It will enable playing virtual drums through air gestures detected by the IMU, with real-time audio synthesis and low latency.
System Overview
The data pipeline in our system relies on multiple communication protocols between the different hardware components. The BNO085 sensor communicates with the Arduino Nano ESP32 using the I²C protocol, with the high-level SHTP protocol layered on top to enable structured transfer of fused motion data. Once the sensor data is received by the ESP32, it is forwarded to the FPGA over an SPI interface, where the FPGA operates as the SPI slave and the ESP32 serves as the SPI master. On the FPGA, the data is processed and packaged into the custom transmission format used for the system.
From there, the FPGA sends the packaged data to the STM32 microcontroller through a second SPI link, again with the FPGA acting as the slave and the STM32 as the master. In the microcontroller, the data is further interpreted to identify which drum corresponds to the detected motion input. The resulting drum-selection signal is then output through a digital-to-analog converter (DAC) to drive a speaker that plays the correct drum sound.
This multi-protocol architecture enables reliable motion data transfer across all computing elements in the system. The correct functionality of each communication stage is demonstrated in the Results section of this report.
Data Flow Diagram
BNO085 Sensor (I2C/SHTP) → Arduino ESP32 → [SPI] → FPGA → [SPI] → STM32 MCU → DAC → Speaker
Communication Protocols: - I2C/SHTP: BNO085 sensor to Arduino ESP32 - SPI: Arduino ESP32 to FPGA (ESP32 = master, FPGA = slave) - SPI: FPGA to STM32 MCU (MCU = master, FPGA = slave) - DAC: STM32 MCU to Speaker (12-bit DAC on PA4)
Hardware Components
BNO085 IMU Sensor
The BNO085 is a 9-axis intelligent IMU that integrates an accelerometer, gyroscope, and magnetometer with an onboard sensor-fusion processor. Instead of exposing raw sensor registers directly, the BNO085 communicates through the Sensor Hub Transport Protocol (SHTP) — a structured, packet-based protocol that runs over standard physical interfaces including I²C, SPI, or UART.
Key Features: - Onboard ARM Cortex-M0+ microcontroller running Bosch’s BSX Lite fusion firmware - Performs sensor fusion (combining accel + gyro + mag) internally - Provides high-level motion data like quaternions and rotation vectors - Offloads computational cost of sensor fusion from host processor - Well-suited for high-performance embedded applications
SHTP Protocol: - Organizes communication into channels (configuration, sensor data, control/status) - Each channel maintains sequence numbers for reliable packet delivery - Packet structure: 4-byte header (length, channel, sequence) + payload (Report ID + sensor data) - Continually sends report packets at configured update rate
Arduino Nano ESP32
The Arduino Nano ESP32 combines the ESP32-S3’s processing power with the Arduino ecosystem, providing: - Integrated Wi-Fi and Bluetooth connectivity - Larger memory capacity for advanced sensor-fusion libraries - Easy integration with Adafruit_BNO08x library
Library Integration: - Uses Adafruit_BNO08x library for high-level BNO085 interface - Abstracts low-level SHTP packet parsing - Simple API calls for initialization, report enabling, and data retrieval - I2C communication with default address (0x4A)
Library: Adafruit_BNO08x
FPGA (UPduino)
The FPGA serves as a data bridge between the Arduino ESP32 and STM32 MCU: - Receives sensor data from ESP32 via SPI (slave mode) - Processes and packages data into custom transmission format - Sends packaged data to STM32 MCU via SPI (slave mode) - Implements dual SPI slave interfaces
FPGA Design Details: - Top-level module: drum_trigger_top.sv - Arduino SPI slave: arduino_spi_slave.sv - MCU SPI slave: spi_slave_mcu.sv - System clock: 3MHz (48MHz HSOSC divided by 16)
STM32 MCU
The STM32L432KC microcontroller: - Receives gesture data from FPGA via SPI (master mode) - Unpacks WAV files from flash memory (parsing RIFF headers and extracting audio samples) - Identifies which drum corresponds to detected motion input - Plays audio samples through 12-bit DAC on PA4 at correct sample rate - Drives speaker to play the correct drum sound
MCU Design Details: - SPI1 master configuration (PB3=SCK, PB5=MOSI, PB4=MISO, PA11=NSS) - DAC output on PA4 - Flash memory for WAV file storage
Communication Protocols
SPI Communication: ESP32 → FPGA
Configuration: - Clock Speed: 100kHz - Mode: SPI Mode 0 (CPOL=0, CPHA=0) - Bit Order: MSB first - CS Pin: D10 (FPGA_SPI_CS)
Packet Format (16 bytes): - Byte 0: Header (0xAA) - Bytes 1-2: Roll (int16_t, MSB first, scaled by 100) - Bytes 3-4: Pitch (int16_t, MSB first, scaled by 100) - Bytes 5-6: Yaw (int16_t, MSB first, scaled by 100) - Bytes 7-8: Gyro X (int16_t, MSB first, scaled by 2000) - Bytes 9-10: Gyro Y (int16_t, MSB first, scaled by 2000) - Bytes 11-12: Gyro Z (int16_t, MSB first, scaled by 2000) - Byte 13: Flags (bit 0 = Euler valid, bit 1 = Gyro valid) - Bytes 14-15: Reserved (0x00)
SPI Communication: FPGA → MCU
Configuration: - Mode: SPI Mode 0 (CPOL=0, CPHA=0) - Bit Order: MSB first - FPGA passes raw 16-byte packet buffer directly to MCU - MCU reads data via SPI master transactions
Results
SPI Data Transmission: ESP32 → FPGA
Validation Using Logic Analyzer
The SPI bus was monitored using a logic analyzer to verify correct data transfer between the Arduino ESP32 and the FPGA. The captured waveform confirms that the expected byte sequence is transmitted from the Arduino to the FPGA when new motion data is available.
Validated Packet Example:
| Parameter | Raw Bytes (Hex) | Value (Decimal) | Interpretation |
|---|---|---|---|
| Header | AA | — | Packet start marker |
| Roll | 03 74 | 0x0374 = 884 | ~88.4° (divide by 10) |
| Pitch | FF 31 | 0xFF31 = –207 | ~–20.7° (divide by 10) |
| Yaw | 1F 0D | 0x1F0D = 7949 | ~794.9° (before wrap-around) |
Results: - Decoded values match Arduino serial monitor output - SPI protocol functioning correctly - Data formatting matches intended system design - Arduino-to-FPGA communication pipeline is reliable
SPI Data Transmission: FPGA → MCU
Validation Using Logic Analyzer
The SPI communication between FPGA and MCU was validated using a logic analyzer, confirming correct data transfer and protocol compliance.
DAC Output
The 12-bit DAC on PA4 successfully outputs audio samples at the correct sample rate, enabling real-time drum sound playback through the speaker.
Schematics & Diagrams
System Block Diagram
TODO: Add complete system block diagram
Please add a high-level system block diagram showing: - BNO085 sensor with I2C connection to Arduino ESP32 - Arduino ESP32 with SPI connection to FPGA - FPGA with SPI connection to STM32 MCU - STM32 MCU with DAC connection to speaker - Data flow arrows indicating direction - All communication protocols labeled (I2C/SHTP, SPI) - Bus widths indicated
Recommended Image Specifications: - Format: PNG or SVG - Size: 1200px width minimum - Location: Save to Images/system-block-diagram.png
Example:
{width="100%" alt="Complete system block diagram"}Pin Connection Diagram
TODO: Add pin connection diagram
Please add a comprehensive pin connection diagram showing: - All interconnections between components - Signal names and pin numbers - Power and ground distribution
Code Repository
TODO: Add GitHub repository URL
View Complete Source Code on GitHub
The repository contains all source code, including: - FPGA Verilog modules (Code/fpga/) - MCU C code (Code/mcu/) - Arduino/ESP32 code (Code/Nuk_Option/Arduino/) - Testbenches and simulation files - Documentation files
Key Source Files
FPGA: - drum_trigger_top.sv - Top-level module - arduino_spi_slave.sv - Arduino SPI slave interface - spi_slave_mcu.sv - MCU SPI slave interface
MCU: - main.c - Main application code - STM32L432KC_SPI.c - SPI driver - WAV file parsing and DAC output code
Arduino: - ARDUINO_SENSOR_BRIDGE.ino - Sensor bridge code using Adafruit_BNO08x library
Photos & Video
Finished Design Photos
TODO: Add photos of the complete system
Please add photos showing: - Complete assembled system - BNO085 sensor connection - FPGA board close-up - MCU board close-up - Arduino/ESP32 setup - Wiring and connections - Speaker and audio output
Recommended Specifications: - Format: JPG or PNG - Size: 1500-2000px width minimum - Location: Save to Images/ directory
Example:
{width="100%" alt="Complete assembled system"}Video Demonstration
TODO: Add video link or embed
Please add your demonstration video showing: - System overview and setup - Air gesture detection - Real-time drum sound playback - Multiple drum sounds being triggered - Low latency demonstration
Video Requirements: - Hosted in the provided Google Drive folder - Named: <LastName1><LastName2><VideoName> (e.g., BellidoStralkaInvisibleDrumSet) - Shows the complete system in operation
Embedding Options: - Google Drive embed (iframe) - Direct link to Google Drive - YouTube/Vimeo embed
Bill of Materials
| Material | Quantity | Cost | Total |
|---|---|---|---|
| BNO085 IMU Sensor | 2 | $24.95 | $49.90 |
| Speaker (8Ω, 4W) | 1 | Stockroom | $0.00 |
| Switches (drums) | 2 | Stockroom | $0.00 |
| Potentiometer | 1 | Stockroom | $0.00 |
| SD Card | 1 | Stockroom | $0.00 |
| TOTAL | $49.90 |
Total estimated cost under $50, excluding development hardware.
References
Acknowledgements
We want to thank Professor Matthew Spencer and staff engineer Xavier Walter for their constant support and help throughout this project.